Overly optimistic prediction results on imbalanced data: a case study of flaws and benefits when applying over-sampling

Information extracted from electrohysterography recordings could potentially prove to be an interesting additional source of information estimate the risk on preterm birth. Recently, a large number studies have reported near-perfect results distinguish between patients that will deliver term or using public resource, called Term/Preterm Electrohysterogram database. However, we argue these are overly optimistic due methodological flaw being made. In this work, focus one specific type flaw: applying over-sampling before partitioning data into mutually exclusive training and testing sets. We show how causes biased two artificial datasets reproduce in which was identified. Moreover, evaluate actual impact predictive performance, when applied prior partitioning, same methodologies related studies, provide realistic view methodologies’ generalization capabilities. make our research reproducible by providing all code under open license.